<!DOCTYPE html>
<html lang="zh-CN">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>RocketMQ消息堆积问题监控与排查指南</title>
    <link href="https://cdn.staticfile.org/font-awesome/6.4.0/css/all.min.css" rel="stylesheet">
    <link href="https://cdn.staticfile.org/tailwindcss/2.2.19/tailwind.min.css" rel="stylesheet">
    <link href="https://fonts.googleapis.com/css2?family=Noto+Serif+SC:wght@400;500;600;700&family=Noto+Sans+SC:wght@300;400;500;700&display=swap" rel="stylesheet">
    <style>
        body {
            font-family: 'Noto Sans SC', Tahoma, Arial, Roboto, "Droid Sans", "Helvetica Neue", "Droid Sans Fallback", "Heiti SC", "Hiragino Sans GB", Simsun, sans-serif;
            color: #333;
            line-height: 1.6;
            background-color: #f9fafb;
        }
        .hero {
            background: linear-gradient(135deg, #1e3a8a 0%, #2563eb 100%);
            color: white;
        }
        .card {
            transition: transform 0.3s ease, box-shadow 0.3s ease;
        }
        .card:hover {
            transform: translateY(-5px);
            box-shadow: 0 10px 25px rgba(0, 0, 0, 0.1);
        }
        .code-block {
            background-color: #2d3748;
            color: #f8fafc;
            border-radius: 0.375rem;
            position: relative;
        }
        .code-header {
            background-color: #1a202c;
            padding: 0.5rem 1rem;
            border-top-left-radius: 0.375rem;
            border-top-right-radius: 0.375rem;
            display: flex;
            justify-content: space-between;
            align-items: center;
        }
        .copy-btn {
            background-color: #4a5568;
            color: white;
            border: none;
            padding: 0.25rem 0.5rem;
            border-radius: 0.25rem;
            cursor: pointer;
            font-size: 0.75rem;
            transition: background-color 0.2s;
        }
        .copy-btn:hover {
            background-color: #2d3748;
        }
        .section-divider {
            border: none;
            height: 1px;
            background: linear-gradient(90deg, rgba(37, 99, 235, 0.1), rgba(37, 99, 235, 0.8), rgba(37, 99, 235, 0.1));
            margin: 3rem 0;
        }
        .tooltip {
            position: relative;
            display: inline-block;
        }
        .tooltip .tooltiptext {
            visibility: hidden;
            width: 200px;
            background-color: #1e3a8a;
            color: #fff;
            text-align: center;
            border-radius: 6px;
            padding: 5px;
            position: absolute;
            z-index: 1;
            bottom: 125%;
            left: 50%;
            margin-left: -100px;
            opacity: 0;
            transition: opacity 0.3s;
        }
        .tooltip:hover .tooltiptext {
            visibility: visible;
            opacity: 1;
        }
        ::selection {
            background: rgba(59, 130, 246, 0.5);
            color: white;
        }
    </style>
</head>
<body>
    <!-- Hero Section -->
    <section class="hero py-20 px-4 md:px-0">
        <div class="container mx-auto max-w-5xl text-center">
            <div class="flex justify-center mb-6">
                <span class="px-4 py-1 bg-white bg-opacity-20 rounded-full text-sm font-semibold inline-flex items-center">
                    <i class="fas fa-rocket mr-2"></i> RocketMQ 专业指南
                </span>
            </div>
            <h1 class="text-4xl md:text-5xl font-bold mb-6 font-serif">消息堆积问题监控与排查</h1>
            <p class="text-xl md:text-2xl opacity-90 max-w-3xl mx-auto mb-8">全面解析RocketMQ消息堆积问题的监控方法、排查流程与调试技巧</p>
            <div class="flex flex-wrap justify-center gap-4">
                <a href="#monitoring" class="px-6 py-3 bg-white text-blue-900 font-medium rounded-lg hover:bg-opacity-90 transition-all">
                    <i class="fas fa-chart-line mr-2"></i>监控方法
                </a>
                <a href="#troubleshooting" class="px-6 py-3 bg-white bg-opacity-20 text-white font-medium rounded-lg hover:bg-opacity-30 transition-all">
                    <i class="fas fa-search mr-2"></i>排查流程
                </a>
                <a href="#tools" class="px-6 py-3 bg-white bg-opacity-20 text-white font-medium rounded-lg hover:bg-opacity-30 transition-all">
                    <i class="fas fa-tools mr-2"></i>调试工具
                </a>
            </div>
        </div>
    </section>

    <!-- Main Content -->
    <main class="container mx-auto max-w-5xl px-4 py-12">
        <!-- Monitoring Section -->
        <section id="monitoring" class="mb-20">
            <div class="flex items-center mb-8">
                <div class="w-2 h-10 bg-blue-600 mr-4"></div>
                <h2 class="text-3xl font-bold font-serif">堆积问题的监控方法</h2>
            </div>
            <p class="text-lg text-gray-700 mb-8">消息堆积是RocketMQ系统中常见的问题，它表示消息的生产速度超过了消费速度，导致消息在Broker中积压。有效的监控方法可以帮助及时发现堆积问题，防止系统性能下降。</p>

            <div class="grid md:grid-cols-2 gap-8 mb-12">
                <div class="card bg-white p-6 rounded-xl shadow-md">
                    <div class="flex items-center mb-4">
                        <div class="w-10 h-10 bg-blue-100 rounded-full flex items-center justify-center mr-4">
                            <i class="fas fa-tachometer-alt text-blue-600"></i>
                        </div>
                        <h3 class="text-xl font-bold">监控指标</h3>
                    </div>
                    <ul class="space-y-4">
                        <li>
                            <h4 class="font-semibold text-blue-800 flex items-center">
                                <i class="fas fa-database mr-2 text-blue-500"></i> 消息堆积量
                            </h4>
                            <p class="text-gray-600 ml-6">最直接的堆积指标，表示未被消费的消息数量，可以按照Topic、消费组、队列等维度进行监控。</p>
                        </li>
                        <li>
                            <h4 class="font-semibold text-blue-800 flex items-center">
                                <i class="fas fa-clock mr-2 text-blue-500"></i> 消费延迟
                            </h4>
                            <p class="text-gray-600 ml-6">表示消息从生产到被消费的时间间隔，延迟增加通常是堆积的前兆。</p>
                        </li>
                        <li>
                            <h4 class="font-semibold text-blue-800 flex items-center">
                                <i class="fas fa-bolt mr-2 text-blue-500"></i> 消费TPS
                            </h4>
                            <p class="text-gray-600 ml-6">每秒消费的消息数量，消费TPS低于生产TPS会导致堆积。</p>
                        </li>
                        <li>
                            <h4 class="font-semibold text-blue-800 flex items-center">
                                <i class="fas fa-users mr-2 text-blue-500"></i> 消费者数量
                            </h4>
                            <p class="text-gray-600 ml-6">活跃消费者的数量，消费者数量减少可能导致堆积。</p>
                        </li>
                        <li>
                            <h4 class="font-semibold text-blue-800 flex items-center">
                                <i class="fas fa-balance-scale mr-2 text-blue-500"></i> 消费者负载
                            </h4>
                            <p class="text-gray-600 ml-6">各个消费者的负载分布情况，不均衡的负载可能导致部分队列堆积。</p>
                        </li>
                    </ul>
                </div>

                <div class="card bg-white p-6 rounded-xl shadow-md">
                    <div class="flex items-center mb-4">
                        <div class="w-10 h-10 bg-blue-100 rounded-full flex items-center justify-center mr-4">
                            <i class="fas fa-tools text-blue-600"></i>
                        </div>
                        <h3 class="text-xl font-bold">监控工具</h3>
                    </div>
                    <div class="space-y-6">
                        <div>
                            <h4 class="font-semibold text-blue-800 flex items-center">
                                <i class="fas fa-tv mr-2 text-blue-500"></i> RocketMQ Dashboard
                            </h4>
                            <p class="text-gray-600 ml-6">RocketMQ官方提供的Web管理控制台，提供消息堆积量、消费进度等可视化监控。</p>
                        </div>
                        <div>
                            <h4 class="font-semibold text-blue-800 flex items-center">
                                <i class="fas fa-terminal mr-2 text-blue-500"></i> RocketMQ Console
                            </h4>
                            <p class="text-gray-600 ml-6">基于命令行的监控工具，提供更详细的监控数据和操作功能。</p>
                        </div>
                        <div>
                            <h4 class="font-semibold text-blue-800 flex items-center">
                                <i class="fas fa-chart-bar mr-2 text-blue-500"></i> JMX监控
                            </h4>
                            <p class="text-gray-600 ml-6">RocketMQ提供JMX接口，可以与Prometheus、Grafana等监控系统集成。</p>
                        </div>
                        <div>
                            <h4 class="font-semibold text-blue-800 flex items-center">
                                <i class="fas fa-code mr-2 text-blue-500"></i> 自定义监控
                            </h4>
                            <p class="text-gray-600 ml-6">基于RocketMQ的Admin API开发自定义监控工具，可以根据业务需求定制监控指标。</p>
                        </div>
                    </div>
                </div>
            </div>

            <div class="mb-12">
                <h3 class="text-2xl font-bold mb-6 font-serif flex items-center">
                    <i class="fas fa-sliders-h mr-3 text-blue-600"></i> 监控策略
                </h3>
                <div class="grid md:grid-cols-2 lg:grid-cols-4 gap-6">
                    <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                        <div class="text-blue-600 mb-3 text-2xl">
                            <i class="fas fa-exclamation-triangle"></i>
                        </div>
                        <h4 class="font-bold mb-2">阈值监控</h4>
                        <p class="text-gray-600 text-sm">设置堆积量的告警阈值，当堆积量超过阈值时触发告警。</p>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                        <div class="text-blue-600 mb-3 text-2xl">
                            <i class="fas fa-chart-line"></i>
                        </div>
                        <h4 class="font-bold mb-2">趋势监控</h4>
                        <p class="text-gray-600 text-sm">监控堆积量的变化趋势，堆积量持续增长是严重问题的信号。</p>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                        <div class="text-blue-600 mb-3 text-2xl">
                            <i class="fas fa-percentage"></i>
                        </div>
                        <h4 class="font-bold mb-2">差异监控</h4>
                        <p class="text-gray-600 text-sm">监控生产速率和消费速率的差异，差异持续扩大表示堆积正在加剧。</p>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                        <div class="text-blue-600 mb-3 text-2xl">
                            <i class="fas fa-calendar-alt"></i>
                        </div>
                        <h4 class="font-bold mb-2">周期性监控</h4>
                        <p class="text-gray-600 text-sm">考虑业务的周期性特点，在业务高峰期加强监控频率。</p>
                    </div>
                </div>
            </div>

            <div class="mb-8">
                <h3 class="text-2xl font-bold mb-4 font-serif">命令行工具示例</h3>
                <div class="code-block mb-6">
                    <div class="code-header">
                        <span>查看消费者状态</span>
                        <button class="copy-btn" onclick="copyCode(this)">复制</button>
                    </div>
                    <pre class="p-4 overflow-x-auto"><code>sh mqadmin consumerStatus -g ConsumerGroupName -n namesrvAddr</code></pre>
                </div>
                <div class="code-block mb-6">
                    <div class="code-header">
                        <span>查看Topic状态</span>
                        <button class="copy-btn" onclick="copyCode(this)">复制</button>
                    </div>
                    <pre class="p-4 overflow-x-auto"><code>sh mqadmin topicStatus -t TopicName -n namesrvAddr</code></pre>
                </div>
                <div class="code-block">
                    <div class="code-header">
                        <span>JMX监控配置</span>
                        <button class="copy-btn" onclick="copyCode(this)">复制</button>
                    </div>
                    <pre class="p-4 overflow-x-auto"><code>-Dcom.rocketmq.client.log.loadconfig=true -Drocketmq.client.logRoot=/path/to/logs
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false</code></pre>
                </div>
            </div>
        </section>

        <hr class="section-divider">

        <!-- Troubleshooting Section -->
        <section id="troubleshooting" class="mb-20">
            <div class="flex items-center mb-8">
                <div class="w-2 h-10 bg-blue-600 mr-4"></div>
                <h2 class="text-3xl font-bold font-serif">堆积原因的排查流程</h2>
            </div>
            <p class="text-lg text-gray-700 mb-8">当发现消息堆积问题时，需要系统性地排查原因，找出瓶颈所在。以下是完整的排查流程：</p>

            <div class="mb-12">
                <h3 class="text-2xl font-bold mb-6 font-serif flex items-center">
                    <i class="fas fa-search mr-3 text-blue-600"></i> 初步分析
                </h3>
                <div class="grid md:grid-cols-3 gap-6 mb-8">
                    <div class="card bg-white p-6 rounded-xl shadow-md">
                        <div class="flex items-center mb-3">
                            <div class="w-8 h-8 bg-blue-100 rounded-full flex items-center justify-center mr-3">
                                <span class="text-blue-600 font-bold">1</span>
                            </div>
                            <h4 class="font-bold">确认堆积范围</h4>
                        </div>
                        <p class="text-gray-600 text-sm">确定是全局堆积还是特定Topic/消费组堆积，检查是所有队列都堆积还是部分队列堆积。</p>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md">
                        <div class="flex items-center mb-3">
                            <div class="w-8 h-8 bg-blue-100 rounded-full flex items-center justify-center mr-3">
                                <span class="text-blue-600 font-bold">2</span>
                            </div>
                            <h4 class="font-bold">查看系统负载</h4>
                        </div>
                        <p class="text-gray-600 text-sm">检查Broker服务器的CPU、内存、磁盘IO、网络等资源使用情况，检查消费者服务器的资源使用情况。</p>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md">
                        <div class="flex items-center mb-3">
                            <div class="w-8 h-8 bg-blue-100 rounded-full flex items-center justify-center mr-3">
                                <span class="text-blue-600 font-bold">3</span>
                            </div>
                            <h4 class="font-bold">检查消费者状态</h4>
                        </div>
                        <p class="text-gray-600 text-sm">确认消费者是否在线，检查消费者数量是否足够，查看消费者的心跳信息和连接状态。</p>
                    </div>
                </div>

                <h3 class="text-2xl font-bold mb-6 font-serif flex items-center">
                    <i class="fas fa-search-plus mr-3 text-blue-600"></i> 深入排查
                </h3>
                <div class="grid md:grid-cols-2 gap-8">
                    <div>
                        <div class="card bg-white p-6 rounded-xl shadow-md mb-6">
                            <h4 class="font-bold text-lg mb-4 flex items-center">
                                <i class="fas fa-user-cog mr-2 text-blue-500"></i> 消费者问题排查
                            </h4>
                            <ul class="space-y-4">
                                <li>
                                    <h5 class="font-semibold text-blue-800">消费速度慢</h5>
                                    <p class="text-gray-600 text-sm">检查消费者的业务处理逻辑是否有性能问题，查看消费者的GC情况。</p>
                                </li>
                                <li>
                                    <h5 class="font-semibold text-blue-800">消费者异常</h5>
                                    <p class="text-gray-600 text-sm">检查消费者的错误日志，查看消费者是否频繁重启。</p>
                                </li>
                                <li>
                                    <h5 class="font-semibold text-blue-800">消费者配置问题</h5>
                                    <p class="text-gray-600 text-sm">检查消费并发度、批量消费配置、拉取配置等。</p>
                                </li>
                            </ul>
                        </div>
                        <div class="card bg-white p-6 rounded-xl shadow-md">
                            <h4 class="font-bold text-lg mb-4 flex items-center">
                                <i class="fas fa-network-wired mr-2 text-blue-500"></i> 网络问题排查
                            </h4>
                            <ul class="space-y-4">
                                <li>
                                    <h5 class="font-semibold text-blue-800">网络连接状态</h5>
                                    <p class="text-gray-600 text-sm">检查网络连接状态，测试网络延迟和丢包率。</p>
                                </li>
                                <li>
                                    <h5 class="font-semibold text-blue-800">网络带宽</h5>
                                    <p class="text-gray-600 text-sm">查看网络带宽使用情况，是否达到瓶颈。</p>
                                </li>
                            </ul>
                        </div>
                    </div>
                    <div>
                        <div class="card bg-white p-6 rounded-xl shadow-md mb-6">
                            <h4 class="font-bold text-lg mb-4 flex items-center">
                                <i class="fas fa-server mr-2 text-blue-500"></i> Broker问题排查
                            </h4>
                            <ul class="space-y-4">
                                <li>
                                    <h5 class="font-semibold text-blue-800">Broker负载过高</h5>
                                    <p class="text-gray-600 text-sm">检查Broker的CPU、内存使用率，查看磁盘IO情况。</p>
                                </li>
                                <li>
                                    <h5 class="font-semibold text-blue-800">Broker配置问题</h5>
                                    <p class="text-gray-600 text-sm">检查Broker的线程池配置、内存配置、磁盘配置。</p>
                                </li>
                                <li>
                                    <h5 class="font-semibold text-blue-800">Broker异常</h5>
                                    <p class="text-gray-600 text-sm">检查Broker的错误日志，查看Broker是否频繁重启。</p>
                                </li>
                            </ul>
                        </div>
                        <div class="card bg-white p-6 rounded-xl shadow-md">
                            <h4 class="font-bold text-lg mb-4 flex items-center">
                                <i class="fas fa-envelope mr-2 text-blue-500"></i> 消息特性分析
                            </h4>
                            <ul class="space-y-4">
                                <li>
                                    <h5 class="font-semibold text-blue-800">消息大小</h5>
                                    <p class="text-gray-600 text-sm">检查消息大小是否异常，是否过大影响消费速度。</p>
                                </li>
                                <li>
                                    <h5 class="font-semibold text-blue-800">生产速率</h5>
                                    <p class="text-gray-600 text-sm">查看消息生产速率是否突增，超出系统处理能力。</p>
                                </li>
                            </ul>
                        </div>
                    </div>
                </div>
            </div>

            <div>
                <h3 class="text-2xl font-bold mb-6 font-serif flex items-center">
                    <i class="fas fa-lightbulb mr-3 text-blue-600"></i> 常见堆积原因
                </h3>
                <div class="grid md:grid-cols-2 lg:grid-cols-3 gap-6">
                    <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                        <div class="flex items-start mb-3">
                            <div class="w-8 h-8 bg-blue-100 rounded-full flex items-center justify-center mr-3 flex-shrink-0">
                                <i class="fas fa-tachometer-alt text-blue-600"></i>
                            </div>
                            <div>
                                <h4 class="font-bold">消费者性能问题</h4>
                                <p class="text-gray-600 text-sm mt-1">消费者业务处理逻辑复杂，处理时间长；与外部系统交互慢；内部资源竞争。</p>
                            </div>
                        </div>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                        <div class="flex items-start mb-3">
                            <div class="w-8 h-8 bg-blue-100 rounded-full flex items-center justify-center mr-3 flex-shrink-0">
                                <i class="fas fa-users text-blue-600"></i>
                            </div>
                            <div>
                                <h4 class="font-bold">消费者数量不足</h4>
                                <p class="text-gray-600 text-sm mt-1">消费者实例数量少于队列数量；动态扩缩容不及时；负载不均衡。</p>
                            </div>
                        </div>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                        <div class="flex items-start mb-3">
                            <div class="w-8 h-8 bg-blue-100 rounded-full flex items-center justify-center mr-3 flex-shrink-0">
                                <i class="fas fa-microchip text-blue-600"></i>
                            </div>
                            <div>
                                <h4 class="font-bold">系统资源瓶颈</h4>
                                <p class="text-gray-600 text-sm mt-1">CPU使用率高；内存不足，频繁GC；磁盘IO瓶颈；网络带宽限制。</p>
                            </div>
                        </div>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                        <div class="flex items-start mb-3">
                            <div class="w-8 h-8 bg-blue-100 rounded-full flex items-center justify-center mr-3 flex-shrink-0">
                                <i class="fas fa-cog text-blue-600"></i>
                            </div>
                            <div>
                                <h4 class="font-bold">消费者配置不合理</h4>
                                <p class="text-gray-600 text-sm mt-1">消费线程数过小；批量消费大小不合理；拉取阈值设置不当。</p>
                            </div>
                        </div>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                        <div class="flex items-start mb-3">
                            <div class="w-8 h-8 bg-blue-100 rounded-full flex items-center justify-center mr-3 flex-shrink-0">
                                <i class="fas fa-arrow-up text-blue-600"></i>
                            </div>
                            <div>
                                <h4 class="font-bold">生产突增</h4>
                                <p class="text-gray-600 text-sm mt-1">业务高峰期；系统异常导致消息重发；历史消息重放。</p>
                            </div>
                        </div>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                        <div class="flex items-start mb-3">
                            <div class="w-8 h-8 bg-blue-100 rounded-full flex items-center justify-center mr-3 flex-shrink-0">
                                <i class="fas fa-exclamation-triangle text-blue-600"></i>
                            </div>
                            <div>
                                <h4 class="font-bold">消费异常</h4>
                                <p class="text-gray-600 text-sm mt-1">消费逻辑异常导致重试；外部依赖异常；消费者频繁重启。</p>
                            </div>
                        </div>
                    </div>
                </div>
            </div>
        </section>

        <hr class="section-divider">

        <!-- Tools Section -->
        <section id="tools" class="mb-20">
            <div class="flex items-center mb-8">
                <div class="w-2 h-10 bg-blue-600 mr-4"></div>
                <h2 class="text-3xl font-bold font-serif">调试工具和命令的使用说明</h2>
            </div>
            <p class="text-lg text-gray-700 mb-8">RocketMQ提供了丰富的工具和命令，用于调试和解决消息堆积问题。以下是常用工具和命令的使用说明：</p>

            <div class="mb-12">
                <h3 class="text-2xl font-bold mb-6 font-serif flex items-center">
                    <i class="fas fa-terminal mr-3 text-blue-600"></i> 命令行工具
                </h3>
                <div class="grid md:grid-cols-2 gap-6">
                    <div class="card bg-white p-6 rounded-xl shadow-md">
                        <h4 class="font-bold text-lg mb-4 flex items-center">
                            <i class="fas fa-user-circle mr-2 text-blue-500"></i> 查看消费者状态
                        </h4>
                        <div class="code-block mb-4">
                            <div class="code-header">
                                <span>命令</span>
                                <button class="copy-btn" onclick="copyCode(this)">复制</button>
                            </div>
                            <pre class="p-4 overflow-x-auto"><code>sh mqadmin consumerStatus -g ConsumerGroupName -n namesrvAddr</code></pre>
                        </div>
                        <div class="bg-gray-100 p-4 rounded-lg">
                            <h5 class="font-semibold mb-2">输出示例：</h5>
                            <pre class="text-sm"><code>Consumer Group: ConsumerGroupName
Topic: TopicName
Broker Name: broker-a
Queue ID: 0
Broker Offset: 1000
Consumer Offset: 800
Diff: 200</code></pre>
                        </div>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md">
                        <h4 class="font-bold text-lg mb-4 flex items-center">
                            <i class="fas fa-list-alt mr-2 text-blue-500"></i> 查看Topic状态
                        </h4>
                        <div class="code-block mb-4">
                            <div class="code-header">
                                <span>命令</span>
                                <button class="copy-btn" onclick="copyCode(this)">复制</button>
                            </div>
                            <pre class="p-4 overflow-x-auto"><code>sh mqadmin topicStatus -t TopicName -n namesrvAddr</code></pre>
                        </div>
                        <div class="bg-gray-100 p-4 rounded-lg">
                            <h5 class="font-semibold mb-2">输出示例：</h5>
                            <pre class="text-sm"><code>Topic: TopicName
Broker Name: broker-a
Queue ID: 0
Min Offset: 0
Max Offset: 1000
Last Updated: 2023-05-01 12:00:00</code></pre>
                        </div>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md">
                        <h4 class="font-bold text-lg mb-4 flex items-center">
                            <i class="fas fa-server mr-2 text-blue-500"></i> 查看Broker状态
                        </h4>
                        <div class="code-block mb-4">
                            <div class="code-header">
                                <span>命令</span>
                                <button class="copy-btn" onclick="copyCode(this)">复制</button>
                            </div>
                            <pre class="p-4 overflow-x-auto"><code>sh mqadmin brokerStatus -b brokerAddr</code></pre>
                        </div>
                        <div class="bg-gray-100 p-4 rounded-lg">
                            <h5 class="font-semibold mb-2">输出示例：</h5>
                            <pre class="text-sm"><code>Broker: broker-a
Version: V4.9.3
BrokerID: 0
Boot Time: 2023-05-01 00:00:00
TPS: 1000</code></pre>
                        </div>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md">
                        <h4 class="font-bold text-lg mb-4 flex items-center">
                            <i class="fas fa-history mr-2 text-blue-500"></i> 重置消费位点
                        </h4>
                        <div class="code-block">
                            <div class="code-header">
                                <span>命令</span>
                                <button class="copy-btn" onclick="copyCode(this)">复制</button>
                            </div>
                            <pre class="p-4 overflow-x-auto"><code>sh mqadmin resetOffsetByTime -t TopicName -g ConsumerGroupName -s timestamp -f force -n namesrvAddr</code></pre>
                        </div>
                    </div>
                </div>
            </div>

            <div class="mb-12">
                <h3 class="text-2xl font-bold mb-6 font-serif flex items-center">
                    <i class="fas fa-tv mr-3 text-blue-600"></i> Web管理工具
                </h3>
                <div class="grid md:grid-cols-2 gap-6">
                    <div class="card bg-white p-6 rounded-xl shadow-md">
                        <h4 class="font-bold text-lg mb-4 flex items-center">
                            <i class="fas fa-users mr-2 text-blue-500"></i> 查看消费者组
                        </h4>
                        <p class="text-gray-600 mb-4">访问 <code class="bg-gray-100 px-2 py-1 rounded">http://&lt;dashboard-address&gt;/consumer</code> 查看消费详情。</p>
                        <div class="bg-blue-50 p-4 rounded-lg border border-blue-200">
                            <h5 class="font-semibold text-blue-800 mb-2">功能：</h5>
                            <ul class="list-disc pl-5 text-blue-700 space-y-1">
                                <li>查看消费进度</li>
                                <li>查看堆积量</li>
                                <li>查看TPS信息</li>
                                <li>查看消费者连接状态</li>
                            </ul>
                        </div>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md">
                        <h4 class="font-bold text-lg mb-4 flex items-center">
                            <i class="fas fa-list mr-2 text-blue-500"></i> 查看Topic
                        </h4>
                        <p class="text-gray-600 mb-4">访问 <code class="bg-gray-100 px-2 py-1 rounded">http://&lt;dashboard-address&gt;/topic</code> 查看Topic详情。</p>
                        <div class="bg-blue-50 p-4 rounded-lg border border-blue-200">
                            <h5 class="font-semibold text-blue-800 mb-2">功能：</h5>
                            <ul class="list-disc pl-5 text-blue-700 space-y-1">
                                <li>查看消息生产情况</li>
                                <li>查看队列分布</li>
                                <li>查看Topic配置</li>
                                <li>查看消息路由信息</li>
                            </ul>
                        </div>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md">
                        <h4 class="font-bold text-lg mb-4 flex items-center">
                            <i class="fas fa-server mr-2 text-blue-500"></i> 查看Broker
                        </h4>
                        <p class="text-gray-600 mb-4">访问 <code class="bg-gray-100 px-2 py-1 rounded">http://&lt;dashboard-address&gt;/broker</code> 查看Broker状态。</p>
                        <div class="bg-blue-50 p-4 rounded-lg border border-blue-200">
                            <h5 class="font-semibold text-blue-800 mb-2">功能：</h5>
                            <ul class="list-disc pl-5 text-blue-700 space-y-1">
                                <li>查看Broker运行状态</li>
                                <li>查看配置信息</li>
                                <li>查看性能指标</li>
                                <li>查看存储情况</li>
                            </ul>
                        </div>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md">
                        <h4 class="font-bold text-lg mb-4 flex items-center">
                            <i class="fas fa-envelope mr-2 text-blue-500"></i> 消息查询
                        </h4>
                        <p class="text-gray-600 mb-4">访问 <code class="bg-gray-100 px-2 py-1 rounded">http://&lt;dashboard-address&gt;/message</code> 查询消息详情。</p>
                        <div class="bg-blue-50 p-4 rounded-lg border border-blue-200">
                            <h5 class="font-semibold text-blue-800 mb-2">功能：</h5>
                            <ul class="list-disc pl-5 text-blue-700 space-y-1">
                                <li>按照Topic查询</li>
                                <li>按照Key查询</li>
                                <li>按照消息ID查询</li>
                                <li>查看消息内容</li>
                            </ul>
                        </div>
                    </div>
                </div>
            </div>

            <div>
                <h3 class="text-2xl font-bold mb-6 font-serif flex items-center">
                    <i class="fas fa-code mr-3 text-blue-600"></i> 编程调试
                </h3>
                <div class="grid md:grid-cols-2 gap-6">
                    <div class="card bg-white p-6 rounded-xl shadow-md">
                        <h4 class="font-bold text-lg mb-4 flex items-center">
                            <i class="fas fa-cogs mr-2 text-blue-500"></i> 使用Admin API
                        </h4>
                        <div class="code-block">
                            <div class="code-header">
                                <span>Java示例</span>
                                <button class="copy-btn" onclick="copyCode(this)">复制</button>
                            </div>
                            <pre class="p-4 overflow-x-auto"><code>DefaultMQAdminExt admin = new DefaultMQAdminExt();
admin.setNamesrvAddr("namesrvAddr");
admin.start();

// 查看消费进度
ConsumeStats stats = admin.examineConsumeStats("ConsumerGroupName");
for (MessageQueue queue : stats.getOffsetTable().keySet()) {
    OffsetWrapper wrapper = stats.getOffsetTable().get(queue);
    long brokerOffset = wrapper.getBrokerOffset();
    long consumerOffset = wrapper.getConsumerOffset();
    long lag = brokerOffset - consumerOffset;
    System.out.printf("Queue: %s, BrokerOffset: %d, ConsumerOffset: %d, Lag: %d%n",
            queue, brokerOffset, consumerOffset, lag);
}

// 查看Topic路由信息
TopicRouteData routeData = admin.examineTopicRouteInfo("TopicName");
List&lt;QueueData&gt; queueDatas = routeData.getQueueDatas();
for (QueueData data : queueDatas) {
    System.out.printf("Broker: %s, ReadQueueNums: %d, WriteQueueNums: %d%n",
            data.getBrokerName(), data.getReadQueueNums(), data.getWriteQueueNums());
}

admin.shutdown();</code></pre>
                        </div>
                    </div>
                    <div class="card bg-white p-6 rounded-xl shadow-md">
                        <h4 class="font-bold text-lg mb-4 flex items-center">
                            <i class="fas fa-user-cog mr-2 text-blue-500"></i> 使用Consumer API调试
                        </h4>
                        <div class="code-block">
                            <div class="code-header">
                                <span>Java示例</span>
                                <button class="copy-btn" onclick="copyCode(this)">复制</button>
                            </div>
                            <pre class="p-4 overflow-x-auto"><code>DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("ConsumerGroupName");
consumer.setNamesrvAddr("namesrvAddr");

// 设置消费起始位置
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_TIMESTAMP);
consumer.setConsumeTimestamp(UtilAll.timeMillisToHumanString3(System.currentTimeMillis()));

// 设置消费线程数
consumer.setConsumeThreadMin(20);
consumer.setConsumeThreadMax(64);

// 设置批量消费
consumer.setConsumeMessageBatchMaxSize(32);

// 设置消费监听器
consumer.registerMessageListener(new MessageListenerConcurrently() {
    @Override
    public ConsumeConcurrentlyStatus consumeMessage(List&lt;MessageExt&gt; msgs, ConsumeConcurrentlyContext context) {
        // 打印消息详情，用于调试
        for (MessageExt msg : msgs) {
            System.out.printf("Message: %s, QueueId: %d, StoreTime: %s, ReconsumeTimes: %d%n",
                    msg.getMsgId(), msg.getQueueId(),
                    UtilAll.timeMillisToHumanString3(msg.getStoreTimestamp()),
                    msg.getReconsumeTimes());
        }
        return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
    }
});

consumer.subscribe("TopicName", "*");
consumer.start();</code></pre>
                        </div>
                    </div>
                </div>
            </div>
        </section>

        <hr class="section-divider">

        <!-- Debugging Best Practices -->
        <section class="mb-20">
            <div class="flex items-center mb-8">
                <div class="w-2 h-10 bg-blue-600 mr-4"></div>
                <h2 class="text-3xl font-bold font-serif">调试最佳实践</h2>
            </div>
            <div class="grid md:grid-cols-2 lg:grid-cols-3 gap-6">
                <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                    <div class="flex items-start mb-3">
                        <div class="w-10 h-10 bg-blue-100 rounded-full flex items-center justify-center mr-3">
                            <i class="fas fa-sitemap text-blue-600"></i>
                        </div>
                        <div>
                            <h4 class="font-bold">分阶段调试</h4>
                            <p class="text-gray-600 text-sm mt-1">先确认堆积范围和严重程度，然后排查消费者问题，再检查Broker和网络问题，最后分析消息特性。</p>
                        </div>
                    </div>
                </div>
                <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                    <div class="flex items-start mb-3">
                        <div class="w-10 h-10 bg-blue-100 rounded-full flex items-center justify-center mr-3">
                            <i class="fas fa-tachometer-alt text-blue-600"></i>
                        </div>
                        <div>
                            <h4 class="font-bold">临时提高消费能力</h4>
                            <p class="text-gray-600 text-sm mt-1">增加消费者实例数量，提高消费线程数，优化消费批量大小，减少消费者负载。</p>
                        </div>
                    </div>
                </div>
                <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                    <div class="flex items-start mb-3">
                        <div class="w-10 h-10 bg-blue-100 rounded-full flex items-center justify-center mr-3">
                            <i class="fas fa-vials text-blue-600"></i>
                        </div>
                        <div>
                            <h4 class="font-bold">消息采样分析</h4>
                            <p class="text-gray-600 text-sm mt-1">抽取部分堆积消息进行分析，检查消息内容、大小、生产时间等特征，寻找共同特点。</p>
                        </div>
                    </div>
                </div>
                <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                    <div class="flex items-start mb-3">
                        <div class="w-10 h-10 bg-blue-100 rounded-full flex items-center justify-center mr-3">
                            <i class="fas fa-flask text-blue-600"></i>
                        </div>
                        <div>
                            <h4 class="font-bold">模拟测试</h4>
                            <p class="text-gray-600 text-sm mt-1">在测试环境复现堆积场景，调整各种参数，观察对堆积的影响，验证解决方案的有效性。</p>
                        </div>
                    </div>
                </div>
                <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                    <div class="flex items-start mb-3">
                        <div class="w-10 h-10 bg-blue-100 rounded-full flex items-center justify-center mr-3">
                            <i class="fas fa-layer-group text-blue-600"></i>
                        </div>
                        <div>
                            <h4 class="font-bold">分批处理堆积</h4>
                            <p class="text-gray-600 text-sm mt-1">对于严重堆积，可以分批次处理，使用resetOffsetByTime命令重置消费位点，逐步消费堆积消息。</p>
                        </div>
                    </div>
                </div>
                <div class="card bg-white p-6 rounded-xl shadow-md hover:border-blue-500 border border-transparent transition-all">
                    <div class="flex items-start mb-3">
                        <div class="w-10 h-10 bg-blue-100 rounded-full flex items-center justify-center mr-3">
                            <i class="fas fa-bell text-blue-600"></i>
                        </div>
                        <div>
                            <h4 class="font-bold">建立监控预警</h4>
                            <p class="text-gray-600 text-sm mt-1">设置合理的堆积阈值，实时监控消费延迟，建立快速响应机制。</p>
                        </div>
                    </div>
                </div>
            </div>
            <div class="mt-10 bg-blue-50 border-l-4 border-blue-500 p-4 rounded-r-lg">
                <div class="flex">
                    <div class="flex-shrink-0">
                        <i class="fas fa-info-circle text-blue-500 text-xl mt-1"></i>
                    </div>
                    <div class="ml-3">
                        <h3 class="text-lg font-bold text-blue-800">系统化解决方案</h3>
                        <p class="text-blue-700">通过系统性的监控、排查和调试，可以有效发现和解决RocketMQ中的消息堆积问题，保证消息系统的稳定运行。</p>
                    </div>
                </div>
            </div>
        </section>
    </main>

    <!-- Footer -->
    <footer class="bg-gray-900 text-white py-10">
        <div class="container mx-auto max-w-5xl px-4 text-center">
            <div class="mb-4">
                <h3 class="text-xl font-bold">技术小馆</h3>
                <p class="text-gray-400 mt-2">专注于技术分享与最佳实践</p>
            </div>
            <div class="text-gray-400">
                <a href="http://www.yuque.com/jtostring" class="hover:text-white transition-colors" target="_blank">
                    <i class="fas fa-book mr-1"></i> www.yuque.com/jtostring
                </a>
            </div>
        </div>
    </footer>

    <script src="https://cdn.jsdelivr.net/npm/mermaid@latest/dist/mermaid.min.js"></script>
    <script>
        // Copy code button functionality
        function copyCode(button) {
            const codeBlock = button.parentElement.nextElementSibling;
            const range = document.createRange();
            range.selectNode(codeBlock);
            window.getSelection().removeAllRanges();
            window.getSelection().addRange(range);
            document.execCommand('copy');
            window.getSelection().removeAllRanges();
            
            const originalText = button.textContent;
            button.textContent = '已复制!';
            button.style.backgroundColor = '#10B981';
            
            setTimeout(() => {
                button.textContent = originalText;
                button.style.backgroundColor = '#4A5568';
            }, 2000);
        }

        // Initialize Mermaid
        mermaid.initialize({
            startOnLoad: true,
            theme: 'default',
            flowchart: {
                useMaxWidth: true,
                htmlLabels: true,
                curve: 'basis'
            }
        });

        // Smooth scrolling for anchor links
        document.querySelectorAll('a[href^="#"]').forEach(anchor => {
            anchor.addEventListener('click', function (e) {
                e.preventDefault();
                document.querySelector(this.getAttribute('href')).scrollIntoView({
                    behavior: 'smooth'
                });
            });
        });
    </script>
</body>
</html>